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PERFORMANCE  IN  NON-FACE-TO-FACE  COLLABORATIVE  INFORMATION 
ENVIRONMENTS 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  is  increasingly  operating  in  a  network  centric  environment  where 
Soldiers  work  together  towards  a  common  goal.  Although  the  Soldiers  are  working 
together,  they  may  be  miles  or  thousands  of  miles  apart  using  digital  systems  to  share 
task  information  and  communications.  This  collaboration  of  distal  Soldiers  each  having 
their  own  duties  and  tasks,  raises  many  questions  including:  how  knowledge  of  other 
collaborative  team  members’  tasks  influences  overall  team  performance;  how  mode  of 
communication  influences  performance,  and  how  different  types  of  communication  are 
associated  with  successful  performance.  The  experimental  research  reported  here 
attempts  to  provide  answers  to  these  important  questions. 

Procedure: 

Twenty-eight  pairs  of  college  students  worked  together  in  pairs  over  a  computer 
network  to  locate  scud  missile  launchers  in  the  computer-based  game  SCUDHunt.  The 
students  operated  from  different  rooms  in  a  research  suite;  they  were  physically  isolated 
from  each  other.  Each  partner  had  a  different  array  of  intelligence  gathering  assets, 
with  different  capabilities  and  reliabilities,  to  employ  to  locate  the  launchers.  After 
completing  a  demographics  questionnaire,  each  isolated  team  member  was  taught  the 
rules  of  the  game  and  the  characteristics  of  their  assets.  After  the  first  training  module, 
the  team  member  completed  a  paper  and  pencil  test  on  the  training  material  and  was 
given  immediate  test  feedback.  Each  team  member  then  received  a  second  training 
module.  For  half  the  teams,  this  was  a  repeat  of  the  first  module  in  which  each  team 
member  learned  about  the  assets  he/she  would  manage  in  the  game;  the  other  half  of 
the  teams  learned  about  their  partner’s  assets  during  this  second  training  module. 

These  conditions  were  called  Own  and  All  respectively.  Again,  each  team  member  was 
given  a  test  on  the  training  material  just  presented  and  given  immediate  feedback. 
Teams  then  practiced  the  game  for  one  turn.  Experimenters  answered  any  questions 
regarding  operation  of  the  game  or  the  assets.  Teams  then  played  two  5-turn  games  of 
SCUDHunt  selecting  the  three  most  likely  launcher  locations  at  the  end  of  each  turn. 

For  one  game  each  team  communicated  via  voice  over  headphones  (Oral)  and  used 
text  chat  (Chat)  via  the  computer  for  the  other  game.  After  each  game,  partners 
completed  a  task  load  index.  The  number  of  launchers  located  on  each  turn  was 
recorded  (quality  score)  as  well  as  the  number  of  map  grid  locations  selected  in 
common  by  each  partner  (shared  situational  awareness). 


Findings: 


Method  of  communication  had  no  significant  (p>.05)  effect  upon  either  the  quality 
or  the  shared  situational  awareness  scores.  The  training  condition  did  influence 
performance  with  the  All  teams  locating  significantly  more  launchers  in  game  2  than 
Own  teams  which  had  a  significantly  higher  level  of  shared  situational  awareness.  The 
Own  teams  also  reported  a  higher  workload  after  game  1 .  Analysis  of  the  text 
messages  indicated  greater  success  in  locating  launchers  in  those  teams  with  the 
highest  frequency  of  messages  in  the  communication  categories  regarding:  game 
situation,  player  status  and  non-task  related/social  interactions. 

These  results  indicate  that  performance  on  collaborative  tasks  benefits  when 
collaborating  participants  are  cross-trained  to  have  a  broader,  system-wide  view  of  the 
entire  situation.  Additionally,  this  cross-training  reduces  the  initial  workload  for  the 
cross-trained  teams.  This  research  also  indicated  identifiable  communication  patterns 
related  to  higher  performance.  Understanding  the  types  of  communication  that  develop 
more  accurate  situational  awareness  provides  insights  into  how  to  train  Soldiers  to 
communicate  effectively  in  non-face-to-face  environments. 

Utilization  and  Dissemination  of  Findings: 

These  findings  can  be  used  by  Army  trainers  and  training  developers  when 
designing  or  conducting  training  for  collaborative  network  centric  distal  operations. 
Emphasis  must  be  placed  on  the  value  of  cross-training  and  effective  communication 
among  the  collaborative  team. 

The  findings  of  this  research  have  been  briefed  and  discussed  at  numerous 
TRADOC,  HEL,  and  JFCOM  meetings  and  conferences. 
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PERFORMANCE  IN  NON-FACE-TO-FACE  COLLABORATIVE 
INFORMATION  ENVIRONMENTS 


Technological  innovations  play  a  paradoxical  role  in  military  transformation.  While 
they  help  to  resolve  existing  battlefield  challenges  on  the  one  hand,  they  invariably 
introduce  new  challenges  on  the  other  hand.  Network-centric  operations  (NCO) 
are  just  such  an  innovation. 


(LTG  W.S.  Wallace,  June  2005,  p.2) 


INTRODUCTION 

Today’s  military  operates  in  a  network-enabled  environment  where  technological 
advances  exploit  gathering  and  sharing  information  to  building  shared-situational 
awareness.  These  technologies  have  transformed  the  nature  of  traditional  collaborative 
work  (Easley,  Devaraj,  and  Crant,  2003).  The  research  in  this  report  helps  us 
understand  what  is  different  in  these  network-enabled  environments. 

Sonnenwald  &  Pierce  (1998)  observed  that  using  technology  to  obtain  and 
process  information  requires  training  not  only  in  human-computer  interaction  but  also  in 
human-human-computer  (collaborative)  interaction.  Warfighters  must  not  only  develop 
their  own  situational  awareness  (SA),  they  must  understand  each  others’  SA  (Pew, 
1995).  This  common  ground  is  what  each  collaboration  participant  assumes  about  the 
others  to  ensure  effective  interactions  (Ross,  2003;  Wellons,  1993).  Communication  is 
key.  Collaborators  must  coordinate  and  share  information  (Hevel,  2002;  Salas,  Prince, 
Baker,  and  Shrestha,  1995;  Sonnenwald  &  Pierce,  1998).  Griffin  and  Reid  caution, 
Warfare  is  not  ‘network  centric’.  It  is  either  ‘people  centric’,  or  it  has  not  centre  at  all. 
(Griffin  &  Reid,  unpublished  manuscript). 

Collaboration  influences  military  operations  at  all  levels.  Technical  interoperability 
is  not  enough  to  produce  the  synchronization  required.  Major  General  Gordon  C.  Nash, 
Commander,  Joint  Warfighting  Center,  Director,  Joint  Training  J7  states: 

.Collaboration  capabilities  can  affect  all  aspects  of  joint  operations. 

. . .  These  capabilities  are  important  to  Department  of  Defense  efforts  to  transform 
the  way  we  plan  and  execute  joint  operations.  To  accomplish  this  transformational 
task,  we  must  improve  collaboration  among  combatant  commands,  Services, 
agencies,  and  multinational  partners.  An  environment  of  collaboration  can 
enable  and  integrate  such  a  cooperative  effort  among  these  organizations  and 
help  the  joint  force  achieve  decision  superiority. 
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Information  sharing  while  working  towards  a  common  goal  is  a  key  aspect  of 
collaboration  (Alberts  2003).  To  gain  insights  into  issues  involving  collaboration  and 
training,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(USARI)  conducted  observations  and  interviews  of  U.S.  Army  personnel  who  operated 
digital  systems  incorporated  into  their  units  (Schaab  &  Dressel,  2003;  Schaab,  Dressel, 
&  Hayes,  2005).  It  soon  became  clear  that  classroom  training  on  how  to  use  digital 
systems  was  not  enough.  Soldiers  understood  that  job  success  requires  an 
understanding  of  how  their  system  interacts  with  other  systems.  Developing  both  a  clear 
sense  of  how  to  collaborate  with  people  operating  those  other  systems  and  an 
appreciation  of  how  important  collaboration  is  in  achieving  and  maintaining  situational 
understanding  requires  experience  in  multiple  training  exercises  that  incorporate  a 
variety  of  scenarios.  In  one  command  center,  Soldiers  actually  place  two  different 
systems  side-by-side  and  cross-train  each  other  to  promote  collaboration.  They 
comprehended  the  need  to  understand  the  interrelationship  between  their  roles.  Such 
opportunities  to  foster  mutual  understanding  become  more  difficult,  of  course,  when 
members  are  dispersed.  Alberts  (2003)  cautions  that,  without  appropriate  training  and 
practice,  the  network-centric  environment  might  actually  increase  the  fog  of  war  rather 
than  provide  superior  situational  understanding.  To  insure  the  latter  result  and  avoid  the 
former,  the  training  side  of  the  military,  in  particular,  needs  to  understand  the  dynamics 
of  this  new  environment,  where  warfighters  increasingly  interact  with  their  peers  and 
leaders  digitally. 

To  improve  the  warfighter’s  ability  to  perform  effectively  in  this  information-rich  and 
changing  environment,  the  military  needs  to  train  personnel  at  all  levels  to  do  essential 
coordination  with  others  across  horizontal  and  vertical  networks.  The  current  research 
examines  characteristics  of  network-collaborative  environments  (e.g.,  personnel 
geographically  dispersed,  different  knowledge  sets,  unacquainted)  and  how  this 
environment  impacts  performance.  A  second  phase  of  this  research  will  examine 
different  techniques  to  train  participants  to  collaborate. 

RESEARCH  CONCEPTS 

Researchers  first  selected  a  game  that  both  allowed  performance,  using  different 
communication  modes  and  knowledge  sets,  to  be  examined  and  the  collection  of 
communication  exchanges.  Descriptions  of  the  practical  features  of  the  game  and 
several  relevant  underlying  research  concepts  follow. 

Research  Venue 

SCUDHunt,  a  game  developed  by  Thought  link,  INC,  was  selected  for  this  research 
on  collaboration  because  it  provided  a  simplified  model  of  the  interplay  of  shared 
awareness  and  communication,  while  permitting  independent  manipulation  of  variables 
thought  to  affect  them.  SCUDHunt  requires  participants  to  (1 )  collaborate  from 
distributed  locations  and  (2)  share  unique  information  from  their  intelligence  assets  for 
optimal  game  performance.  The  goal  of  the  game  is  simple,  to  locate  three  SCUD 
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missile  launchers  on  a  map.  The  game  requires  geographically-dispersed  players  to 
collaborate  while  executing  digital  tasks  in  order  to  achieve  a  shared  goal. 

Cognitive  Task  Analysis 

The  research  included  a  cognitive  task  analysis  of  this  game  to  identify  critical  points 
where  collaboration  would  be  beneficial  (Ross,  2003).  In  general,  players  need  to 
communicate  planning  strategies  and  to  share  gathered  information  in  order  to  perform 
effectively.  The  collaboration  areas  identified  were: 

Coordinating  deployment:  Players  discuss  where  best  to  place  their  assets  on  the 
map  grid,  with  the  goals  of  (1 )  maximizing  coverage  of  the  area  remaining  to  be 
searched,  and  (2)  using  certain  assets  to  verify  the  results  of  earlier  searches; 

Interpreting  results:  Players  discuss  the  reliability  of  reports  from  different 
intelligence-gathering  assets,  leading  to  a  determination  of  the  likelihood  that  a  SCUD 
launcher  is  at  any  particular  location.  This  involves  interpretation  of  results  from  the 
current  turn,  as  well  as  integration  of  findings  from  previous  searches. 

Common  Knowledge 

This  situation  of  networked  individuals  who  have  shared  goals  but  unique  roles  and 
responsibilities  raises  a  new  question  for  research  on  collaboration  at  a  distance:  To 
what  extent  does  knowledge  about  a  partner’s  role  influence  an  individual’s 
performance  effectiveness?  Knowledge  about  others’  roles,  responsibilities,  and  job 
requirements  has  been  termed  "interpositional  knowledge."  One  effective  training 
strategy  for  increasing  interpositional  knowledge  among  team  members  is  cross-training 
(Blickensderfer,  Cannon-Bowers,  &  Salas,  1998). 

Volpe,  Cannon-Bowers,  Salas,  and  Spector  (1996)  defined  cross-training  as  a 
strategy  where  “each  team  member  is  trained  on  the  tasks,  duties,  and  responsibilities 
of  his  or  her  fellow  team  members”  (p.  87).  This  involves  having  team  members 
understand  and  sometimes  practice  each  other’s  skills.  The  Volpe  et  al.  (1996)  initial 
research  on  cross-training,  as  well  as  an  extension  (Cannon-Bowers,  Salas,  & 
Blickensderfer,  1998),  showed  that  those  team  members  who  received  cross-training 
were  better  able  to  anticipate  each  other’s  needs,  shared  more  information,  and  were 
more  successful  in  task  performance.  Additional  research  has  found  that  cross-training 
and  a  common  understanding  of  roles  contributes  to  shared  mental  model  development, 
effective  communication,  and  improved  coordination  (McCann,  Baranski,  Thompson,  & 
Pigeau,  2000;  Marks,  Sabella,  Burke,  &  Zaccaro,  2002). 

Similarly,  U.S.  Army  trainers  found  that  Soldiers  needed  an  understanding  of  how 
their  digital  system  interacted  and  complemented  other  Army  systems  (Schaab  & 
Dressel,  2003;  Schaab,  Dressel  &  Hayes,  2005).  Computer  systems  training  has  been 
modified  to  include  an  introductory  module  on  the  unique  contributions  of  each  system 
and  how  they  work  together  to  achieve  optimal  situational  understanding. 
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To  investigate  such  issues,  cross-training  (called  the  All  condition)  versus  intensive 
training  in  one  role  (called  the  Own  condition)  was  included  as  a  variable  in  this 
research.  Players  received  either  a  double  dose  of  training  on  the  tasks  they  would 
perform  (deployment  and  interpretation  of  their  own  assets)  or  a  single  dose  of  training 
on  both  their  tasks  and  on  the  tasks  of  their  partner;  this  is  referred  to  as  the  All  training 
condition. 

Communication  Patterns  and  Mode 

Recent  research  has  shown  that  communication  content  (Urban,  Weaver,  Bowers,  & 
Rhodenizer,  1996)  can  influence  team  coordination  and  performance.  To  investigate 
these  relationships,  researchers  explored  how  the  type  of  communication  during  a 
collaboration  effort  was  associated  with  success.  A  two  tier  communication 
categorization  was  used  to  analyze  text  chat  communication  between  players  (see 
Appendix  A).  The  first  tier  consisted  of  7  types  of  communication  actions  while  the 
second  tier  considered  the  game  related  object  of  the  first  tier  action.  For  example,  a 
message  could  be  categorized  as  a  request  for  information  (action)  regarding  asset 
capability  (object  of  action).  More  specifically,  consider  the  communication,  “How 
reliable  is  your  satellite  in  detecting  a  SCUD?”  First  tier  (action)  is  a  request  for 
information  and  the  second  tier  (object  of  the  action)  is  asset  capability. 

In  addition,  communication  mode  was  manipulated.  All  pairs  wore  headsets  that 
allowed  oral  communication  during  one  of  the  two  games  they  played;  during  the  other 
game,  participants  communicated  by  sending  typed  messages  via  an  on-screen  "chat" 
box.  For  a  random  half  of  the  pairs,  the  "chat"  game  came  first.  The  data  analyzed 
included  measures  of  the  types  and  frequency  of  communication  between  participants 
in  the  "chat"  game  and  game  performance. 

METHOD 

Participants 

Fifty-six  undergraduate  students,  31  females  and  25  males,  received  course  credit 
for  three  hours  of  participation.  Participants  ranged  in  age  from  18  to  35  years,  with  the 
overwhelming  majority  between  18  to  25  years.  Five  participants  had  military 
experience.  Participants  were  paired  for  the  research.  Forty-two  individuals  forming  21 
pairs  had  never  met.  The  remaining  participants  were  acquaintances  or  friends. 

Instruments 

Questionnaire:  At  the  beginning  of  the  session,  participants  completed  a  questionnaire 
that  requested  demographic  information  (e.g.,  gender,  age,  and  military  experience)  and 
computer  experience. 

Workload:  The  NASA  Task  Load  Index  (TLX)  (Hart  &  Staveland,  1988)  is  a  multi¬ 
dimensional  rating  too!  that  provides  workload  scores  based  on  six  subscales:  Mental 
Demands,  Physical  Demands,  Temporal  Demands,  Own  Performance,  Effort,  and 
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Frustration.  A  definition  of  each  subscale  is  provided  in  Appendix  B.  The  NASA  TLX 
was  administered  to  each  participant  after  completing  the  first  and  second  game  of 
SCUDHunt. 

Game  Description:  The  SCUDHunt  game  presents  players  with  the  mission  of 
determining  where  -  on  a  five-by-five  grid  board  representing  the  map  of  a  hostile 
country  -  the  launchers  for  SCUD  missiles  are  located.  Participants  are  told  that  there 
are  three  SCUD  launchers,  each  in  a  different  fixed  location  among  the  25  squares  on 
the  board.  On  each  of  five  turns,  participants  deploy  intelligence-gathering  assets  (for 
example,  a  reconnaissance  satellite  or  a  team  of  Navy  Seals),  receive  reports  from 
those  assets,  and  create  a  “strike  plan”  (to  be  sent  to  their  fictional  commander) 
indicating  their  independent  best  guesses  based  on  all  the  information  received  both 
from  that  turn  and  previous  turns  as  to  the  SCUD  launcher  locations.  They  are  told  that 
the  final  strike  plans  -  after  the  fifth  turn  -  will  be  used  by  their  commander  to  direct  an 
attack  on  the  SCUD  launchers,  and  they  are  given  the  results  of  this  final  strike  plan  in 
terms  of  which  bombed  location  held  a  now-destroyed  launcher.  Participants  control 
either  air  or  ground  assets  (see  Figure  1 )  with  each  asset  having  unique  capabilities  and 
returning  intelligence  reports  of  different  reliabilities.  For  example,  eyes  on  target  reports 
from  Human  Intelligence  assets  would  be  more  reliable  than  reports  generated  from 
satellites  where  sensors  must  interpret  images  from  great  distances.  This  game  then  is 
a  representation  of  a  situation  in  which  Soldiers  would  use  digital  systems  to  execute 
tasks  requiring  collaboration  for  successful  performance. 


Air  Control 


Ground  Control 


Satellite  UAV 


Manned  Aircraft 


Figurel.  Symbols  for  air  and  ground  assets  controlled  by  participants. 
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Measures  generated  as  the  game  is  played  are  1 )  the  number  of  SCUD  launcher 
locations  identified,  called  quality  score,  2)  the  degree  to  which  the  two  participants  on 
a  team  chose  the  same  grid  squares  or  location  in  their  independent  strike  plans,  called 
shared-situational  awareness,  3)  the  number  of  text  chat  communications  taking 
place,  4)  the  categorized  content  of  the  text  chat,  5)  measures  of  subjective  workload 
reported  on  the  NASA  TLX,  and  6)  responses  to  questionnaire  items  on  demographics 
and  computer  experiences. 

Design.  Three  primary  independent  variables  for  this  experiment  were  training 
condition,  mode  of  communication  and  communication  patterns.  All  versus  Own 
training  conditions  involved  training  on  the  characteristic  of  the  information-gathering 
assets  used  in  the  SCUDHunt  game.  Every  participant  received,  as  their  first  training 
module,  an  explanation  of  the  characteristics  of  the  assets  they  would  be  controlling. 
Half  of  the  pairs  (the  Own  condition)  received  a  second  exposure  to  the  same  asset 
training;  the  other  half  (the  All  condition)  received  training  in  which  each  participant 
learned  the  characteristics  of  the  assets  to  be  controlled  by  that  participant's  partner. 

Mode  of  communication  (Oral  or  Chat)  was  analyzed  to  determine  if  communication 
mode  influenced  game  performance. 

Communication  patterns,  as  described  in  Appendix  A,  were  also  evaluated  to 
determine  if  the  frequency  of  certain  types  of  communication  were  associated  with 
higher  quality  scores  and  increased  shared-situational  understanding. 

Procedure.  Upon  arrival  at  the  laboratory,  individual  participants  were  ushered  into 
separate  rooms  where  they  read  and  signed  a  standard  consent  form  describing  the 
experiment  and  their  rights  as  participants.  This  was  followed  by  each  participant 
completing  a  questionnaire  on  demographics  and  their  experience  with  computers  and 
computer  games.  Researchers  then  explained  that  the  experiment  involved  participants 
playing  a  computer  game  with  a  partner  who  was  located  in  another  room. 

Several  computer-based  training  modules  were  then  presented  on  1)  the  overall 
aspects  of  playing  the  SCUDHunt  game  and  2)  the  characteristics  of  the  information¬ 
gathering  assets  used  in  playing  the  game.  Following  each  training  module,  participants 
took  paper  and  pencil  quizzes  on  the  material  just  presented  and  were  given  immediate 
corrective  feedback,  if  necessary,  to  ensure  that  they  understood  how  to  play  the  game 
and  the  capabilities  of  their  assets.  After  this  training,  the  pair  played  a  one-turn  practice 
game,  to  ensure  that  the  mechanics  of  playing  the  game  were  understood.  After  the 
experimenters  answered  any  question  the  participants  might  have,  the  pair  played  two 
complete  five-turn  games  of  SCUDHunt.  During  these  games,  data  were  automatically 
collected  on  1 )  the  messages  participants  sent  to  each  other,  2)  the  degree  to  which 
grid  squares  chosen  as  targets  in  the  "strike  plans"  (submitted  at  the  end  of  each  turn) 
were  identical  for  the  two  members  of  the  pair,  and  3)  the  number  of  those  chosen 
target  squares  that  actually  contained  missile  launchers. 
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RESULTS  and  DISCUSSION 


Performance  by  Mode  of  Communication 

All  participants  played  the  game  in  each  of  the  two  communication  modes  (Oral 
or  Chat)  with  the  order  of  presentation  balanced  across  pairs.  Analyses  of  variance 
revealed  no  significant  effect  (p>.05)  of  mode  of  communication  upon  either  the  number 
of  Scud  launcher  identified  (the  quality  score)  or  the  concordance  of  best  guesses  of  the 
partners  (shared  situational  awareness). 

Performance  by  Training  Condition 

Participants  were  trained  either  on  both  their  tasks  and  their  partner’s  tasks  (All 
condition)  or  they  were  trained  twice  on  their  own  tasks  (Own  condition). 


During  the  first  of  the  two  games  played,  no  differences  in  performance  were 
seen  between  those  trained  in  the  All  or  the  Own  conditions  in  either  quality  score  or 
shared-situational  awareness.  In  the  second  game,  participants  in  the  All  condition 
located  significantly  more  SCUD  launchers  than  did  those  in  the  Own  condition  (see 
Figure  2),  but  those  in  the  Own  condition  had  significantly  higher  levels  of  shared- 
situational  awareness  (see  Figure  3).  This  means  that  participants  who  were  cross- 
trained  in  both  their  role  and  in  their  partner’s  role  were  more  successful  in  locating 
SCUD  launchers.  Participants  trained  solely  in  their  role  achieved  higher  levels  of 
agreement  with  their  partner  on  where  they  thought  the  SCUD  launchers  were  located, 
but  were  wrong  more  frequently  than  those  who  were  cross-trained. 


Figure  2.  Quality  score  for  participants  trained  in  the  All  versus  the  Own  condition. 
Participants  in  the  All  condition  located  significantly  more  SCUD  launchers  in  Game  2 
than  those  in  the  Own  condition  (F  (1 , 54)  =  4.435,  p<.05). 
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Figure  3.  Shared-situational  awareness  for  participants  trained  in  the  Own  versus  the 
All  condition.  Participants  in  the  Own  reported  more  SCUD  launcher  location  in 
common  in  Game  2  than  did  those  in  the  All  condition  (F  (1 , 54)  =  4.49,  p<. 05).  No 
locations  in  common  =0;  All  3  locations  in  common=2 

Workload. 

Participants  completed  the  NASA  TLX  after  game  1  and  game  2.  During  game  1 , 
those  in  the  Own  condition  reported  higher  levels  of  time  demand  (F  (1 ,  53)  =  1 1 .1 62,  p 
<  .05)  and  frustration  (F  (1 ,  53)  =  6.275,  p  <.05)  than  those  in  the  All  condition  (see 
Figure  4).  No  differences  in  workload  were  reported  during  Game  2.  This  suggests  that 
participants  in  the  Own  condition  initially  experienced  higher  levels  of  workload  than 
those  trained  in  the  All  condition. 
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Figure  4.  Comparison  of  workload  in  the  Own  and  All  conditions.  After  Game  1 , 
participants  in  the  Own  condition  reported  significant  higher  levels  of  Time  Demand  and 
Frustration  than  those  in  the  All  condition. 

Age,  Computer  Experience,  and  Gender  Differences 

Fifty-one  of  the  56  participants  (91  percent)  were  25  years  old  or  younger  with  none 
of  the  participants  over  age  25  reporting  “a  lot"  of  experience  with  either  computer/video 
games  or  web-based  gaming.  Although  differences  in  gaming  experience  were 
reported,  no  significant  differences  in  performance  between  “older”  and  younger  players 
were  found.  Comparisons  between  older  and  younger  participants  are  tenuous 
because  these  groups  are  dissimilar  in  size  and  the  “older”  group  was  between  26-35 
years  of  age. 

Relationships  between  computer  experience  in  gaming  and  gender  further  our 
understanding  of  how  these  variables  interact.  Most  participants  used  computers  for 
email,  searching  the  internet  and  instant  messaging  (see  Figure  5).  Thirteen  of  56 
participants  (23  percent)  reported  “a  lot”  of  experience  with  computer/video  games, 
while  only  6  of  the  56  (1 1  percent)  had  “a  lot”  of  experience  with  web-based  gaming.  A 
significant  correlation  was  found  (r=  -.264,  p<. 05)  between  experience  with  web-based 
gaming  and  the  number  of  SCUD  launchers  located  in  the  first  game  played. 
Participants  with  more  web-based  gaming  experience  actually  performed  worse  during 
the  first  game.  This  difference  was  not  seen  in  the  second  game.  It  is  possible  that  prior 
gaming  experience  initially  resulted  in  negative  transfer  to  the  new  game.  Additional 
research  would  be  needed  to  understand  this  relationship  more  fully.  No  other 
significant  correlations  were  found  between  any  of  the  types  of  computer  experience 
reported  and  performance. 
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Figure  5.  Survey  results  describing  participants’  computer  experience. 

Significant  gender  differences  were  found,  with  males  reporting  more  experience  in 
the  categories  of  web-based  gaming  (X,2  -i,  3=  6.47)  and  computer/video  games  (X,2 .05, 3= 
15.37)  than  did  females  (see  Tables  1  and  2)  Despite  these  significant  differences  in 
gaming  experience,  no  difference  in  performance,  either  in  level  of  shared-situational 
awareness  or  number  of  SCUD  launchers  located,  was  found  between  males  and 
females.  Although  males  engage  in  more  computer  gaming  activities  than  females, 
these  experiences  did  not  lead  to  better  performance  on  this  game. 

Additionally,  when  looking  at  males  only,  no  differences  in  game  performances  were 
found  as  a  function  of  relative  gaming  experience.  These  findings  imply  that  regardless 
of  prior  gaming  experience,  people  can  benefit  from  training  that  incorporates  serious 
gaming  activities. 

Table  1.  Experience  with  Web-based  Gaming 


Males  (Percentage) 

Females  (Percentage) 

None 

20 

45 

A  little 

44 

36 

Some 

16 

16 

A  lot 

20 

3 
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Table  2.  Experience  with  Computer/Video  Games 


Males  (Percentage) 

Females  (Percentage) 

None 

0 

13 

A  little 

16 

45 

Some 

40 

36 

A  lot 

44 

6 

Also,  no  significant  differences  were  found  in  quality  scores  or  shared-situational 
awareness  based  on  computer  experience,  military  background,  or  acquaintance  with 
partner. 

Communication  Patterns 

Players  with  a  higher  frequency  of  communications  in  the  categories  of  game 
situation,  player  status,  and  non-task  related/social,  identified  a  significantly  greater 
number  of  SCUD  launchers  (Game  Situation  F  (3,  51 )  =  4.78,  p<. 05;  Player  Status  ( F 
(3,  51)  =  6.43,  p<. 05;  Non-task  Related/Social  F  ( 3,  51)  =  8.166,  p<. 05).  Examples  of 
these  types  of  communication  patterns  follow. 

Game  Situation:  The  specific  game  context  the  players  have  created  by  their  play  of 
the  current  game. 

“Not  much  going  on  in  E2.”  or 

“Grid  square  C3  looks  like  a  possible  target.” 


Turn.  1  Phese:  Search  Plan  History  of  Search  Results 

Status:  Foi  the  UAV,  click  on  any  contiguous  5  cells. 


fa  j  <  I  ►{ 

l  5  | 


It 


Pcrtt  Blank  Game  Boards 
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Non-task  Related  Social:  Social  remarks  not  game  related. 


“Anyone 
know  a 
good  pizza 
place?” 


Interestingly,  the  second  highest  number  of  communications  in  each  of  these 
three  categories  occurred  when  only  one  Scud  launcher  was  located.  This  difference 
was  significant  only  for  the  Game  Situation  category  (see  Figure  6) 


Figure  6.  Number  of  communications  exchanged  compared  to  number  of  SCUDs 
located.  Significantly  more  communications  took  place  when  all  three  SCUD  launchers 
were  located. 
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CONCLUSIONS 


These  results  demonstrate  that  collaborative  tasks  benefit  when  collaborating 
participants  are  cross  trained  to  have  a  broader,  system-wide  view  of  the  entire 
situation.  Participants  trained  on  both  their  role  and  their  partner’s  role  (All  condition) 
were  more  successful  in  locating  SCUD  launcher  locations  and,  in  Game  1,  did  so  with 
lower  levels  of  Time  Demand  and  Frustration  than  found  in  those  trained  solely  on  their 
role  (Own  condition).  In  addition,  while  those  in  the  Own  condition  identified  more 
suspected  SCUD  launcher  locations  in  common  (shared-  situational  awareness)  than 
those  in  the  All  condition,  these  concurrences  were  more  likely  to  be  incorrect  locations. 

Training  Soldiers  of  the  21st  century  to  effectively  communicate  non-face-to-face  in 
order  to  develop  accurate  shared-situational  awareness  is  essential  in  our  transforming 
military.  This  research  implies  that  Soldiers  achieve  higher  levels  of  accurate  situational 
awareness  when  they  understand  each  other’s  role.  Without  this  shared  understanding, 
Soldiers  may  agree  on  what  is  taking  place,  but  this  agreement  is  more  likely  to  be 
incorrect. 

Participants  with  high  levels  of  gaming  experience  performed  similarly  to  those  with 
little  or  no  experience.  Males  and  females  performed  equally  well  despite  males 
reporting  significantly  more  experience  with  gaming  than  did  females.  When  looking  at 
males  only,  those  with  high  levels  of  gaming  experience  did  not  perform  any  differently 
than  those  with  little  or  no  experience. 

Gaming,  which  was  used  in  this  research,  is  being  investigated  as  a  training  tool 
because  it  may  motivate  trainees  to  train  more  frequently  and  for  longer  periods. 
Although  additional  research  is  needed  to  understand  how  to  maximize  the  training 
benefits  of  gaming,  the  current  research  suggests  that  learning  benefits  of  serious 
gaming  may  apply  to  both  gamers  and  non-gamers. 

Identifiable  communication  patterns  are  related  to  performance.  Better 
performers  communicated  more  often.  Communication  categories  related  to  higher 
performance  include  player  status,  game  situation,  and  non-game  related  social. 

Understanding  the  types  of  communication  that  develop  more  accurate 
situational  awareness  provides  insights  into  how  to  train  Soldiers  to  communicate 
effectively  in  non-face-to-face  environments.  Good  performers  keep  their  partner 
informed  of  their  status  and  the  current  situation.  Interestingly,  better  performers 
engaged  in  higher  level  of  non-game  related  social  interaction.  Perhaps  these  social 
communications  aid  in  building  understanding  and  trust. 
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Appendix  A 

Communications  Category  Definitions 

1st  Tier  (Action) 

Request  information:  To  ask  game-related  questions. 

Response  to  information  request:  To  answer  game-related  questions. 

Share  information  -  unsolicited:  To  provide  game-related  information  without  being 
asked. 

Direct  actions:  To  dictate  the  action  of  the  partner. 

WILCO:  To  comply  with  the  directed  action  recommended. 

Non-task  related  (social):  Social  remarks  directed  either  toward  the  play  of  the  game 
(game  related)  or  something  other  than  the  game  (non-game  related). 

Equipment  and  Technical  Problems:  Refers  to  the  operation  of  the  experimental 
apparatus  rather  than  the  play  of  the  game. 

2nd  Tier  (Object) 

Asset  capability  refers  to  how  the  asset  can  be  moved  (deployed)  and  the  reliability  of 
its  findings  (search  results). 

-Can  your  assets  be  placed  individually? 

-Manned  aircraft,  they  are  excellent  at  detecting? 

Player  status  refers  to  the  condition  of  the  player:  is  the  player  ready,  has  the  player 
deployed  assets,  has  the  search  plan  been  submitted. 

-Already  submitted. 

-There,  I  submitted  my  strike  plan. 

-My  SpecOps  is  very  accurate. 

Game  situation  refers  to  the  specific  game  context  the  players  have  created  by  their 
play  of  the  current  game. 

-D1  looks  like  a  target 

-That’s  a  place  where  there’s  not  much  going  on 

Search  strategy  refers  to  the  plan  of  how  assets  can  be  deployed;  i.e.,  placed  on  the 
grid,  to  the  best  advantage  for  target  detection. 

-I  covered  2  sides  of  the  square 
-Where  are  you  searching? 

-For  next  turn,  which  cells  do  you  want  to  take? 


17 


Strike  plan  refers  to  the  plan  of  selecting  which  grid  squares  are  most  likely  to  contain 
targets  as  a  function  of  previous  searches. 

-/  will  strike  Cl. 

-Where  are  you  thinking  about  striking? 

Game  rules/operation  refer  to  the  general  rules  which  govern  the  play  of  the  game 
-This  next  turn  is  our  last. 

-It’s  only  the  last  strike  that  matters. 


Note:  A  communication  may  contain  several  components  each  of  which  needs  to  be 
coded;  however,  each  component  will  have  only  one  code. 
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Appendix  B 


RATING  SCALE  DEFINITIONS 

Title 

Endpoints 

Descriptions 

MENTAL  DEMAND 

Low/High 

How  much  mental  and  perceptual  activity 
was  required  (e.g.,  thinking,  deciding, 
calculating,  remembering,  looking, 
searching,  etc.)?  Was  the  task  easy  or 
demanding,  simple  or  complex,  exacting  or 
forgiving? 

PHYSICAL 

DEMAND 

Low/High 

How  much  physical  activity  was  required 
(e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or 
demanding,  slow  or  brisk,  slack  or 
strenuous,  restful  or  laborious? 

TEMPORAL 

DEMAND 

Low/High 

How  much  time  pressure  did  you  feel  due  to 
the  rate  or  pace  at  which  the  tasks  or  task 
elements  occurred?  Was  the  pace  slow  and 
leisurely  or  rapid  and  frantic? 

EFFORT 

Low/High 

How  hard  did  you  have  to  work  (mentally 
and  physically)  to  accomplish  your  level  of 
performance? 

PERFORMANCE 

Good/Poor 

How  successful  do  you  think  you  were  in 
accomplishing  the  goals  of  the  task  set  by 
the  experimenter  (or  yourself)?  How 
satisfied  were  you  with  your  performance  in 
accomplishing  these  goals? 

FRUSTRATION 

LEVEL 

Low/High 

How  insecure,  discouraged,  irritated, 
stressed  and  annoyed  versus  secure, 
gratified,  content,  relaxed  and  complacent 
did  you  feel  during  the  task? 
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